Immersive virtual musical instrument

An immersive virtual musical instrument, or immersive virtual environment for music and sound, represents sound processes and their parameters as 3D entities of a virtual reality so that they can be perceived not only through auditory feedback but also visually in 3D and possibly through tactile as well as haptic feedback, using 3D interface metaphors consisting of interaction techniques such as navigation, selection and manipulation (NSM)[1]. It builds on the trend in electronic musical instruments to develop new ways to control sound and perform music such as explored in conferences like NIME.

State of the art

Florent Berthaut[2] created a variety of 3D reactive widgets involving novel representations of musical events and sound, that required a special 3D input device to interact with them using adapted 3D interaction techniques.

Jared Bott[3] created an environment that used 3D spatial control techniques as used in known musical instruments, with symbolic 2D visual representation of musical events.

Richard Polfreman[4] made a 3D virtual environment for musical composition with visual representations of musical and sound data similar to 2D composition environments but placed in a 3D space.

Leonel Valbom[5] created a 3D immersive virtual environment with visual 3D representations of musical events and audio spatialization with which could be interacted using NSM interaction techniques.

Teemu Mäki-Patola[6] explored interaction metaphors based on existing musical instruments as seen in his Virtual Xylophone, Virtual Membrane, and Virtual Air Guitar implementations.

Axel Mulder[7] explored the sculpting interaction metaphor by creating a 3D virtual environment that allowed interaction with abstract deformable shapes, such as a sheet and a sphere, which parameters were mapped to sound effects in innovative ways. The work focused on proving the technical feasibility of 3D virtual musical instruments. Gestural control was based on 3D object manipulation such as a subset of prehension.

Early work was done by Jaron Lanier with his Chromatophoria band and separately by Niko Bolas who developed the Soundsculpt Toolkit, a software interface that allows the world of music to communicate with the graphical elements of virtual reality.

References

  1. ^ [Doug Bowman, Ernst Kruijff, Joseph LaViola, Ivan Poupyrev, Wolfgang Stuerzlinger. 3D User Interfaces: Design, Implementation, Usability (2010). Course Notes of the 28th International Conference on Human Factors in Computing Systems. ACM, New York, NY, USA.]
  2. ^ [F. Berthaut, M. Desainte-Catherine, and M. Hachet. DRILE: an immersive environment for hierarchical live-looping. In Proceedings of the 2010 International Conference on New Interfaces for Musical Expression (NIME10), Sydney, NSW, Australia, pages 192–197, 2010.]
  3. ^ [Jared N. Bott, James G. Crowley, and Joseph J. LaViola. 2009. One Man Band: A 3D Gestural Interface for Collaborative Music Creation. In Proceedings of the 2009 IEEE Virtual Reality Conference (VR '09). IEEE Computer Society, Washington, DC, USA, 273-274.]
  4. ^ [R. Polfreman. Frameworks 3d: composition in the third dimension. pages 226–229, Pittsburgh, USA, Carnegie Mellon University, 2009.]
  5. ^ [L Valbom, A Marcos (2007) An Immersive Musical Instrument Prototype IEEE Computer Graphics ans Applications 27: 4. 14-18]
  6. ^ [T. Maaki-Patola, J. Laitinen, A. Kanerva, and T. Takala. Experiments with virtual reality instruments. In Proceedings of the 2005 International Conference on New Interfaces for Musical Expression (NIME05), Vancouver, BC, Canada, 2005.]
  7. ^ [A.G.E. Mulder. Design of virtual three-dimensional instruments for sound control. PhD thesis, Simon Fraser University, Canada, 1998.]

External links